We envision a future where tunnel construction will be undertaken by swarming robots to achieve higher levels of sustainability, automation and efficiency. A core technological enabler is digital reconstruction of the structure being constructed to be able to monitor grout quality and track grout progress. A promising surveying technique involves robots with onboard ground penetrating radar (GPR) travelling inside pipes embedded in the ground. Whilst GPR is a popular technology, applications have tended to focus on identifying defects in roads, tunnels, and bridges, with limited research on detecting buried structural objects around pipes. This study introduces an end-to-end solution for reconstructing grout maps outside pipes using in-pipe GPR. A deep learning algorithm with a classical U-Net architecture is adopted to learn the relationship between circle-scanning radar images and the grout maps. To train the U-Net model and validate the feasibility of the proposed solution, a synthetic dataset containing 2500 images of varying grout sizes, shapes, and positions was constructed using the software gprMax. Predictions on unseen test simulations demonstrate that the proposed solution can provide accurate reconstruction of the grout map outside pipes in all directions.